Add QueueManager & improve TaskQueue by cwillisf · Pull Request #393 · scratchfoundation/scratch-editor

cwillisf · 2025-12-04T18:04:30Z

Resolves

Supports an upcoming scratch-storage change that will resolve scratchfoundation/scratch-gui#7111 and POD-238

Proposed Changes

Main change: in task-herder, add QueueManager, a way to conveniently manage multiple throttling queues.

Supporting changes:

Add tests for the new QueueManager
Fix and simplify the concurrency logic in TaskQueue
Support vendor chunks in scratch-storage, which helps when using npm link scratch-storage
Add {willReadFrequently: true} to the Canvas.getContext('2d') call in the bitmap load path

Reason for Changes

The upcoming scratch-storage change will use this new QueueManager to manage per-host throttling queues.
The previous concurrency logic didn't provide back pressure from the concurrency limiter
- Without that back pressure, the client needs to make extra effort to avoid growing the queue unbounded.
- It's also not consistent with the rate-limiting behavior.
- Removing the p-limit dependency is a nice side effect. (There's nothing wrong with p-limit, but fewer dependencies is better.)
The willReadFrequently change:
- Matches our actual intent with the canvas/context
- Fixes a Chrome console warning
- Fixes a Chrome bug(?) that corrupted buffers when the asset download allowed concurrency > 5
- Might improve performance

Test Coverage

The new QueueManager is covered by new tests. Existing tests cover everything else.

Copilot

Pull request overview

This PR adds a QueueManager class to manage multiple throttling queues and refactors TaskQueue to remove the p-limit dependency by implementing custom concurrency control. The changes also optimize canvas operations by adding willReadFrequently: true to 2D context creation.

Introduces QueueManager for convenient management of multiple per-host throttling queues
Refactors TaskQueue concurrency logic to provide proper back pressure and remove external dependency
Adds canvas optimization flags to fix Chrome warnings and potential buffer corruption

Reviewed changes

Copilot reviewed 8 out of 9 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
packages/task-herder/src/QueueManager.ts	New class for managing multiple named task queues with shared/individual configurations
packages/task-herder/test/manager.test.ts	Comprehensive test coverage for QueueManager API including create, get, and getOrCreate methods
packages/task-herder/src/index.ts	Exports the new QueueManager class
packages/task-herder/src/TaskQueue.ts	Refactored concurrency control with custom implementation, removed p-limit dependency, added options getter
packages/task-herder/package.json	Removed p-limit dependency
package-lock.json	Removed p-limit and yocto-queue dependencies from task-herder
packages/scratch-vm/src/import/load-costume.js	Added willReadFrequently flag to canvas 2D context for bitmap loading
packages/scratch-render/src/BitmapSkin.js	Added willReadFrequently flag with explanatory comment for WebGL texture data extraction
packages/scratch-gui/webpack.config.js	Added vendor chunks copying for scratch-storage npm link support

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

packages/task-herder/src/TaskQueue.ts

packages/task-herder/src/QueueManager.ts

github-actions · 2025-12-04T18:15:08Z